Add experimental uv resolver for Python lockfile generation#23212
Open
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Open
Add experimental uv resolver for Python lockfile generation#23212Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Conversation
Add `[python].lockfile_resolver = "uv"` option that uses `uv pip compile` to pre-resolve pinned requirements before running `pex lock create --no-transitive`. This provides 3-4x faster lockfile generation on cold cache by leveraging uv's PubGrub resolver. All lock styles are supported including `universal` (via `uv pip compile --universal`). Closes pantsbuild#20679 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
Hi @Affanmir, thanks for this contribution. I have some questions about the background, motivation, and creation of this change. It is unusual to receive a PR that directly competes with an existing, pending PR by another contributor that is already going through code review. We like to keep things collegial and friendly here, and so in this case it might have been appropriate to first discuss your change with the author of that PR, and with the maintainers of the project. It would be helpful if you could jump on Slack, introduce yourself there, and give us a bit of background about your use of Pants, your motivation for this change, and how you created it using Claude. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[python].lockfile_resolver = "uv"option (default:pex) for faster lockfile generationuv pip compileto pre-resolve pinned requirements, thenpex lock create --no-transitiveto materialize the PEX lockfileuniversal(viauv pip compile --universal)[uv].args_for_lockfile_resolvefor passthrough flags (e.g.,--index-strategy unsafe-first-match)Motivation
Issue: #20679 — faster, more ergonomic lockfile generation by leveraging uv's PubGrub resolver.
This builds on the uv PEX builder from #23197 (merged) and is inspired by the approach in #22949, with key improvements:
--universal[uv].args(general)[uv].args_for_lockfile_resolve(specific)Benchmarks
Environment: Apple M4, macOS 15.x (Darwin 24.6.0), arm64
Dataset: 87 top-level requirements (Trio test deps + web/data/cloud ecosystem) → 169 resolved packages
Tool: hyperfine v1.20.0
Methodology: Same dataset structure as uv's own benchmarks (Trio project requirements)
Resolver-only comparison (no Pants overhead)
Isolates the dependency resolution step —
uv pip compilevspex lock create:pex lock create(pip)uv pip compileEnd-to-end Pants comparison
Full
pants generate-lockfiles --resolve=...(includes engine init, interpreter discovery, PEX CLI bootstrap):lockfile_resolver = "pex"lockfile_resolver = "uv"Note: End-to-end improvement is smaller because Pants engine overhead (~20s) is constant. The uv resolver also shows much lower variance (±0.8s vs ±13.7s), providing more consistent lockfile generation times.
Correctness
Both resolvers produce equivalent lockfiles — 168/169 packages identical, with minor differences in optional/conditional dependency selection (
greenletvstomli), which is expected behavior between different resolvers.Reproduction
Disclaimers
--no-pantsdused for all measurements (no daemon memoization)How it works
Usage
Current limitations
complete_platformsoverrides/sources/excludesCode changes
subsystems/setup.pyLockfileResolverenum +lockfile_resolveroptionsubsystems/uv.pyargs_for_lockfile_resolvepassthrough + updatedDownloadedUvgoals/lockfile.pygenerate_lockfile()goals/lockfile_test.pylockfiles.mdx2.32.x.mdTest plan
test_strip_named_repo— unit test for index URL stripping helpertest_uv_resolver_rejects_complete_platforms— validates error on unsupported complete_platformstest_uv_resolver_strict_requires_single_python— validates error on multi-version stricttest_uv_resolver_rejects_overrides— validates error on unsupported overridestest_uv_resolver_strict_generates_valid_lockfile— E2E: generates valid PEX lockfile with uv + stricttest_uv_resolver_universal_generates_valid_lockfile— E2E: generates valid PEX lockfile with uv + universalAI disclosure
This PR was created with extensive use of Claude Code (Anthropic's CLI agent). Claude explored the Pants codebase, designed the implementation approach, wrote the code/tests/docs, and ran the benchmarks. All changes were reviewed and test runs verified by the author.
🤖 Generated with Claude Code